{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5767ab9f-1ec6-4668-a47f-d076e3f77387",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5f2deaa-75c9-41a7-a3c0-fbd3f8b64320",
   "metadata": {},
   "source": [
    "# **Evaluating multiple RAG pipelines with RAGElo**\n",
    "Unlike other LLM and RAG evaluation frameworks that try to evaluate every individual LLM answer individually, RAGElo focuses on comparing **pairs** of answers in an Elo-style tournament.\n",
    "\n",
    "The idea is that, without a golden answer, LLMs can't really judge any individual answer in isolation. Rather, analyzing if answer A is better than answer B is a more reliable metric of quality, and makes it easier to decide if a new RAG pipeline is better than another.\n",
    "\n",
    "RAGElo works in three steps when evaluating a RAG pipeline\n",
    "1. Gather all documents retrieved by all agents and annotate their relevance to the user's query.\n",
    "2. For each question, generate \"_games_\" between agents, asking the judging LLM to analyze if one agent is better than another, rather than assigning individual scores to each answer.\n",
    "3. With these games, compute the Elo score for each agent, creating a final ranking of agents.\n",
    "\n",
    "Importantly, RAGElo is **agnostic** to your pipeline. Meaning, it will not directly call your RAG system, and can work with any framework or pipeline you use. When used as a library, as we do here, we should create the `Query` objects with your agent's answers, as we show below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e5d43c-5c46-4955-8a93-a4f30ff6cd34",
   "metadata": {},
   "source": [
    "## 1. Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b6a54813-ad03-480d-bf36-e280a456a53c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/acamara/.pyenv/versions/3.10.12/envs/ragelo310/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import glob\n",
    "import os\n",
    "import pickle\n",
    "import random\n",
    "from getpass import getpass\n",
    "\n",
    "import openai\n",
    "import pandas as pd\n",
    "\n",
    "from ragelo import (\n",
    "    Query,\n",
    "    get_agent_ranker,\n",
    "    get_answer_evaluator,\n",
    "    get_llm_provider,\n",
    "    get_retrieval_evaluator,\n",
    ")\n",
    "from ragelo.types.configurations import (\n",
    "    DomainExpertEvaluatorConfig,\n",
    "    PairwiseDomainExpertEvaluatorConfig,\n",
    ")\n",
    "from ragelo.utils import load_answers_from_multiple_csvs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a38530-f6ef-41b2-930a-0cf3c83f0e2d",
   "metadata": {},
   "source": [
    "## 2. Setup openai key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "755414cd-5b99-4cb6-89ac-e12aad257f33",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not (openai_api_key := os.environ.get(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "openai.api_key = openai_api_key\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f2a083-236a-4e50-800e-0c215e4c36f3",
   "metadata": {},
   "source": [
    "## 3.Load the answers generated by your RAG pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e0abde0e-e382-40ac-996d-c7fdfd874fe4",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder = \"../data/\"\n",
    "csvs = glob.glob(f\"{data_folder}rag_response_*.csv\")\n",
    "\n",
    "queries = load_answers_from_multiple_csvs(\n",
    "    csvs,  # A list of all the CSVs with the answers produced by our RAG pipelinesZZZ\n",
    "    query_text_col=\"question\",  # this tells RAGelo which column of the CSV has the query itself.\n",
    ")\n",
    "query_ids = {q.query: q.qid for q in queries}\n",
    "query_dict = {q.qid: q for q in queries}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8862faa9-2dd7-4995-83c2-173573e581f5",
   "metadata": {},
   "source": [
    "### Parse the documents into the queries\n",
    "\n",
    "Note that the retrieved documents are not tied to a single answer, but to the query itself. In a RAG system, it means that, if the pipeline did not retrieve a relevant document, but another did, the one that didn't performed _worst_ than the one that did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a2b13efc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_docs(raw_docs) -> list[tuple[str, str]]:\n",
    "    docs = raw_docs.split(\"\\n\")\n",
    "    documents = []\n",
    "    for d in docs:\n",
    "        doc_text = d.split(\"document:\", maxsplit=1)[1]\n",
    "        doc_source = d.split(\"source:\", maxsplit=1)[1]\n",
    "        documents.append((doc_source, doc_text))\n",
    "    return documents\n",
    "\n",
    "\n",
    "for csv in csvs:\n",
    "    df = pd.read_csv(csv)\n",
    "    for i, row in df.iterrows():\n",
    "        query_id = query_ids[row[\"question\"]]\n",
    "        answer = row[\"answer\"]\n",
    "        docs = parse_docs(row[\"contexts\"])\n",
    "        query = query_dict[query_id]\n",
    "        for doc_source, doc_text in docs:\n",
    "            query.add_retrieved_doc(\n",
    "                doc_text,\n",
    "                doc_source,  # Here, we use the source as the id of the documents\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f2f56a",
   "metadata": {},
   "source": [
    "## 4. Prepare the Evaluators\n",
    "RAGElo uses _evaluators_ as judges. We will instantiate a **retrieval evaluator**, an **answer evaluator** and an **agents ranker** with their corresponding settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "62b6f1ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The LLM provider can be shared accross all evaluators.\n",
    "llm_provider = get_llm_provider(\"openai\", model_name=\"gpt-4o\", max_tokens=2048)\n",
    "\n",
    "\n",
    "# The DomainExpertEvaluator mimics a persona that is an expert in a given field.\n",
    "# The evaluator will add this persona to the prompt when evaluating the answers.\n",
    "retrieval_evaluator_config = DomainExpertEvaluatorConfig(\n",
    "    expert_in=\"the details of how to better use the Qdrant vector database and vector search engine\",\n",
    "    company=\"Qdrant\",\n",
    "    n_processes=20,  # How many threads to use when evaluating the retrieved documents. Will do that many parallel calls to OpenAI.\n",
    ")\n",
    "\n",
    "retrieval_evaluator = get_retrieval_evaluator(\n",
    "    llm_provider=llm_provider, config=retrieval_evaluator_config\n",
    ")\n",
    "\n",
    "\n",
    "# The PairwiseDomainExpertEvaluator is an Answer evaluator similar to the one above, but evaluates pairs of answers.\n",
    "answer_evaluator_config = PairwiseDomainExpertEvaluatorConfig(\n",
    "    expert_in=\"the details of how to better use the Qdrant vector database and vector search engine\",\n",
    "    company=\"Qdrant\",\n",
    "    n_processes=20,\n",
    "    bidirectional=True,  # LLMs may have a positional bias. So, evaluate games where both agents are on the left and on the right.\n",
    "    n_games_per_query=20,  # The maximum number of games per query to generate. In this case, 20 random games between agents will be evaluated.\n",
    "    document_relevance_threshold=2,  # The minimum relevance score for a document to be considered relevant. Documents with a score lower than this will not be considered in the evaluation.\n",
    ")\n",
    "\n",
    "\n",
    "answer_evaluator = get_answer_evaluator(\n",
    "    llm_provider=llm_provider, config=answer_evaluator_config\n",
    ")\n",
    "\n",
    "# The Elo ranker doesn't need an LLM. Here, we instantiate it with the basic settings.\n",
    "# We can also instantiate evaluators and agents by passing the configuration directly to the get_agent_ranker or get_*__evaluator functions:\n",
    "elo_ranker = get_agent_ranker(\n",
    "    \"elo\",\n",
    "    verbose=True,  # We want to see the final ranking, not only write to an output file.\n",
    "    k=32,  # The k-factor for the Elo ranking algorithm\n",
    "    initial_score=1000,  # Initial score for the agents. This will be updated after each game.\n",
    "    rounds=1000, # Number of tournaments to play\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6b598d-dd9c-454a-b339-cbd3171b3d4d",
   "metadata": {},
   "source": [
    "## 5. Call the evaluators\n",
    "Now, we actually run the evaluators. \n",
    "\n",
    "Note that, as we go, we are adding information to each `Query` object, instead of just dumping everything into CSVs or JSON files. This is by design. The `Query` object is also a Pydantic model, so it can be easily serialized into JSON by calling `query.model_dump_json(<path>)` or pickled by calling `pickle.dumps(query)`.\n",
    "This also avoids re-evaluating the same document multiple times for the same query. The evaluator will also not re-evaluate a query (or answer) that was already evaluated, unless the `force` parameter is set to `True` on its configurations.\n",
    "\n",
    "As the focus of RAGElo is to evaluate the quality of RAG pipelines, the __retrieval__ component is extremely important, and the answers of the agents are evaluated based not only on the quality of the documents they have retrieved, but the quality of all the documents retrieved by any agent. The intuition here is that if there are many relevant documents in the corpus, potentially retrieved by other agents, the evaluation should take these into account, even if a specific agent did not retrieve them.\n",
    "\n",
    "When evaluating a (pair of) answer(s), the LLM will be able to see all the relevant documents retrieved by all agents, and will be able to compare the quality of the answers based on the quality of _all_ the relevant documents retrieved by any agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e04d30de",
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.exists(\"queries.pkl\"):\n",
    "    queries = pickle.load(open(\"queries.pkl\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "482ebe28",
   "metadata": {},
   "outputs": [],
   "source": [
    "queries = retrieval_evaluator.batch_evaluate(queries) # Evaluate all the retrieved documents for all the queries\n",
    "queries = answer_evaluator.batch_evaluate(queries) # As the evaluator is a PairwiseDomainExpertEvaluator, it will create random pairs of agent's answers for the same query and evaluate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "351cfb84",
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(queries, open(\"queries.pkl\", \"wb\")) # Save the results to disk to avoid re-evaluating everything if something goes wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52d2b2e3",
   "metadata": {},
   "source": [
    "### Let's see what is happening under the hood\n",
    "\n",
    "- A query object contain the query itself, the documents retrieved by all the agents, the answers generated by each agent and the pairwise games that the AnswerEvaluator generated.\n",
    "- A Document has three objects that can be evaluated: retrieved documents, agent answers and pairwise games. \n",
    "- Each evaluable object has an `evaluation` object that, after being evaluated, will contain the raw LLM output for the object and a parsed version of it, according to the evaluator's settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fac013dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🔎 The query object:\n",
      "\t💬 Query text: \"What is difference between scalar and product quantization?\"\n",
      "\t📚 7 retrieved documents by all agents\n",
      "\t📊 Average relevance score of the retrieved documents : 1.2857142857142858\n",
      "\t🕵️ 6 Agents answered the query\n",
      "\t🏆 20 games were evaluated for this query\n",
      "--------------------------------------------------------------------------------\n",
      "📜 The document object:\n",
      "\t📄 Document text: \"--- title: \"Qdrant under the hood: Scalar Quantization\" short_description: \"Scalar Quantization is a\" (...)\n",
      "--------------------------------------------------------------------------------\n",
      "📈 Document's evaluation:\n",
      "\t💭 LLM's reasoning for the evaluation: \"Given the user query \"What is the difference between scalar and product quantization?\" and the provi\" (...)\n",
      "\t💯 Document's relevance score (between 0 and 2): 1\n",
      "--------------------------------------------------------------------------------\n",
      "🆚 Pairwise games played:\n",
      "\tGame between agents 🕵️rag_response_512_3_reranked 🆚 🕵️rag_response_512_5\n",
      "\t💭 LLM's reasoning for the evaluation: \"Both Assistant A and Assistant B provide explanations of scalar quantization and product quantizatio (...)\"\n",
      "\t💯 Game's winner: rag_response_512_5\n"
     ]
    }
   ],
   "source": [
    "\n",
    "query = random.choice(queries)\n",
    "\n",
    "print(\"🔎 The query object:\")\n",
    "print(f'\\t💬 Query text: \"{query.query}\"')\n",
    "print(f\"\\t📚 {len(query.retrieved_docs)} retrieved documents by all agents\")\n",
    "average_relevance = sum([int(d.evaluation.answer) for d in query.retrieved_docs]) / len(\n",
    "    query.retrieved_docs\n",
    ")\n",
    "print(f\"\\t📊 Average relevance score of the retrieved documents : {average_relevance}\")\n",
    "print(f\"\\t🕵️ {len(query.answers)} Agents answered the query\")\n",
    "print(f\"\\t🏆 {len(query.pairwise_games)} games were evaluated for this query\")\n",
    "document = random.choice(query.retrieved_docs)\n",
    "print(\"-\" * 80)\n",
    "print(\"📜 The document object:\")\n",
    "print(f'\\t📄 Document text: \"{document.text[:100]}\" (...)')\n",
    "print(\"-\" * 80)\n",
    "print(\"📈 Document's evaluation:\")\n",
    "document_evaluation = document.evaluation\n",
    "print(\n",
    "    f'\\t💭 LLM\\'s reasoning for the evaluation: \"{document_evaluation.raw_answer[:100]}\" (...)'\n",
    ")\n",
    "print(\n",
    "    f\"\\t💯 Document's relevance score (between 0 and 2): {document_evaluation.answer}\"\n",
    ")\n",
    "print(\"-\" * 80)\n",
    "print(\"🆚 Pairwise games played:\")\n",
    "game = random.choice(query.pairwise_games)\n",
    "print(\n",
    "    f\"\\tGame between agents 🕵️{game.agent_a_answer.agent} 🆚 🕵️{game.agent_b_answer.agent}\"\n",
    ")\n",
    "print(\n",
    "    f'\\t💭 LLM\\'s reasoning for the evaluation: \"{game.evaluation.raw_answer[:100]} (...)\"'\n",
    ")\n",
    "best_agent = game.evaluation.answer\n",
    "if best_agent == \"A\":\n",
    "    best_agent = game.agent_a_answer.agent\n",
    "elif best_agent == \"B\":\n",
    "    best_agent = game.agent_b_answer.agent\n",
    "elif best_agent == \"C\":\n",
    "    best_agent = \"TIE\"\n",
    "print(f\"\\t💯 Game's winner: {best_agent}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c81c16f0-fc6d-42f9-b817-2f7e7e721d28",
   "metadata": {},
   "source": [
    "## 6. Rank the agents\n",
    "Based on the results of the games played, we now run the Elo ranker to determine which agent wins the tournament.\n",
    "\n",
    "If we re-run the tournament multiple times, small variations may happen. Therefore, we re-run the tournament multiple times and average the results to get a more stable ranking. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fa3f0eb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------- Agent Scores by Elo Agent Ranker -------\n",
      "rag_response_512_3_reranked: 1006.0\n",
      "rag_response_512_5: 997.0\n",
      "rag_response_512_3: 988.0\n",
      "rag_response_512_5_reranked: 970.0\n",
      "rag_response_512_4_reranked: 936.0\n",
      "rag_response_512_4: 904.0\n"
     ]
    }
   ],
   "source": [
    "elo_ranker.run(queries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8a217f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
